gh-143120: pixi builds for free-threading and TSAN#142872
gh-143120: pixi builds for free-threading and TSAN#142872kumaraditya303 merged 40 commits intopython:mainfrom
Conversation
| script: | ||
| file: ../build.sh | ||
| env: | ||
| PYTHON_VARIANT: "free-threading" |
There was a problem hiding this comment.
copy-paste of asan/recipe.yaml, except this one line
There was a problem hiding this comment.
copy-paste of asan/pixi.toml
Tools/pixi-packages/tsan/recipe.yaml
Outdated
| script: | ||
| file: ../build.sh | ||
| env: | ||
| PYTHON_VARIANT: "tsan" |
There was a problem hiding this comment.
copy-paste of asan/recipe.yaml, except this one line
no clue sorry I haven't seen this before |
crusaderky
changed the title
|
The tsan crash is due to the default $ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32 # too high
$ sudo sysctl vm.mmap_rnd_bits=28 # reduce it
vm.mmap_rnd_bits = 28
|
hugovk
changed the title
|
There's a lot of copy/paste here, which can make maintenance harder. Does pixi have some concept of code reuse or parametrisation? Maybe not the best example, but something like GitHub Actions' reusable workflows? |
|
@lucascolley is there anything we can do right now to reduce the duplication here? Or does that require fixes in pixi? |
|
As mentioned in the README already, this is blocked on prefix-dev/pixi#4599. |
ngoldbaum
left a comment
ngoldbaum
left a comment
There was a problem hiding this comment.
I opened an issue which is needed to make this mergeable. I'm also playing with a NumPy recipe based on this.
crusaderky
changed the title
|
Ready for final review |
|
Your hdf5 recipe is still not using fd6ca6d. Can we please stop the reverting back and forth and debug why the correct commit from cpython is not being used? |
|
For the sake of test velocity and thoroughness, I've created https://github.com/crusaderky/cpython-pixi @isuruf the latest macos-15-intel changes let the python build complete; thank you. Pending
|
@hugovk would you be able to advise? |
@isuruf I appreciate your frustration. I'm frustrated too by this issue. I'm fairly sure that the last CI run I pointed to shows the effect of the libcompiler-rt changes - because I checked - but I'm fallible, so I'll do it again this time making extra sure not to muddle any evidence afterwards. For the sake of velocity and clarity I've moved my CI to https://github.com/crusaderky/cpython-pixi |
Let's not add symlinks to the repo, they're harder to use on Windows and make cross-platform support generally more complex. |
|
The issue was that - if: c_compiler == "gcc" and "san" in variant
then:
- libsanitizer
- if: c_compiler == "clang" and "san" in variant
then:
- libcompiler-rtLinux happened to work because, at least on ubuntu-latest and on my desktop, libasan.so and libtsan.so are installed by default system-wide. I've opened an improvement ticket upstream: prefix-dev/pixi#5393 |
|
I've removed the symlinks and replaced them with physical copies + a sync script. |
|
Gentle ping. |
|
trying this out downstream — I get a solve failure for an env that includes the ASAN Python on this branch, ASAN NumPy on main, removing the |
ASAN NumPy on main points to a specific commit of CPython main. Could you try repointing numpy to numpy/numpy#30510 ? [dependencies]
python.git = "https://github.com/crusaderky/cpython"
python.rev = "tsan"
python.subdirectory = "Tools/pixi-packages/asan"
numpy.git = "https://github.com/ngoldbaum/numpy"
numpy.rev = "tsan-ft-pixi"
numpy.subdirectory = "pixi-packages/asan" |
|
Anyway your problem is that you have an upper pin for python in pythran:
but this recipe builds Python 3.15. Alternatively you can fork pythran and just remove the pin and hope it was just out of an abundance of caution. |
ah thanks, makes sense. Using pythran from git source may get around the issue in the short-term |
kumaraditya303
left a comment
kumaraditya303
left a comment
There was a problem hiding this comment.
I tested the tsan+freethreading pixi config with the following and it worked:
[workspace]
channels = ["conda-forge"]
name = "cpython-pixi-test"
platforms = ["linux-64"]
preview = ["pixi-build"]
[dependencies]
python = { git = "https://github.com/crusaderky/cpython", rev = "tsan", subdirectory = "Tools/pixi-packages/tsan-freethreading" }I'll merge this by tomorrow if there are no objections.
| - Using a single `pixi.toml` and `recipe.yaml` for all package variants is blocked on https://github.com/prefix-dev/pixi/issues/4599 | ||
| - A workaround can be removed from the build script once https://github.com/prefix-dev/rattler-build/issues/2012 is resolved | ||
| - Using a single `pixi.toml` and `recipe.yaml` for all package variants is blocked on | ||
| [pixi-build-backends#532](https://github.com/prefix-dev/pixi-build-backends/pull/532) |
There was a problem hiding this comment.
No need to update the README.md right now, but noting this repo has been archived and the new PR is at prefix-dev/pixi#5364.
| ``` | ||
| To fix it, try reducing `mmap_rnd_bits`: | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
Tip for next time: bash for when there's only commands, and console for when there's commands with output. Compare:
$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32 # too high for TSan
$ sudo sysctl vm.mmap_rnd_bits=28 # reduce it
vm.mmap_rnd_bits = 28$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32 # too high for TSan
$ sudo sysctl vm.mmap_rnd_bits=28 # reduce it
vm.mmap_rnd_bits = 286 participants
Follow-up to #142469
freethreading: compiles with--disable-giltsan-freethreading: compiles with--disable-gil --with-thread-sanitizerrecipe.yamlandpixi.tomlall identical symlinksmacos-15-intelgithub runners)Tested in CI on https://github.com/crusaderky/hdf5-pixi